Skip to content

refactor(compilation): extract ccache into a swappable extension#122

Draft
iav wants to merge 5 commits into
mainfrom
refactor/ccache-as-extension
Draft

refactor(compilation): extract ccache into a swappable extension#122
iav wants to merge 5 commits into
mainfrom
refactor/ccache-as-extension

Conversation

@iav
Copy link
Copy Markdown
Owner

@iav iav commented May 16, 2026

Goal

Move ccache out of core compilation into a swappable extension. After this PR a parallel cache backend (sccache, distcc-style, distributed cache) can ship as a sibling extension without touching core.

A concrete near-term motivation: a future extensions/sccache.sh would let CI workflows running on GitHub-hosted runners route their compile-cache straight into the native GitHub Actions cache (SCCACHE_GHA_ENABLED=on, via mozilla-actions/sccache-action) — 10 GB free per repository, low-latency on the runner's local region. The same extension framework also covers self-hosted runners (S3/R2/MinIO bucket, WebDAV endpoint, Redis-on-disk, ...) without core changes — backend choice is purely an env-var concern per workflow.

What changes (vs armbian/build:main)

New files

  • extensions/ccache.sh — implements compile_prepare_vars, compile_wrapper_pre/_post, and {kernel,uboot,atf,crust}_make_config hooks.
  • lib/functions/compilation/compile-wrapper.shdo_with_compile_wrapper + compile_wrapper_pre/_post hooks.

Removed

  • lib/functions/compilation/ccache.sh (logic moved into the extension).

Modified

  • lib/functions/compilation/atf.sh — adds atf_make_config hook; replaces hardcoded ccache with ${CCACHE} (latent bug: USE_CCACHE=no was still dragging ccache into ATF).
  • lib/functions/compilation/crust.sh — adds crust_make_config hook; drops dead binutils_flags_crust="" declaration (preexisting SC2034 since 2023).
  • lib/functions/compilation/{kernel,uboot}.sh — compile phase wrapped in do_with_compile_wrapper.
  • lib/functions/configuration/compilation-config.sh — dispatches compile_prepare_vars instead of inlining configure-ccache; resets stale CCACHE between artifacts; warns if USE_CCACHE is set in userpatches/lib.config (sourced too late for enable_extension).
  • lib/functions/configuration/main-config.sh — auto-enables the ccache extension when USE_CCACHE=yes, PRIVATE_CCACHE=yes, or ENABLE_EXTENSIONS=ccache-remote, before initialize_extension_manager.
  • lib/functions/general/extensions.sh — adds extension_list_normalized helper (used by the auto-enable shim and per-extension mutex checks).
  • lib/library-functions.sh — regenerated.

User-visible behavior

Unchanged. USE_CCACHE=yes (default for kernel/u-boot) and USE_CCACHE=no behave exactly as before; ENABLE_EXTENSIONS=ccache-remote implicitly pulls in ccache, matching the pre-refactor coupling.

Test plan

Built on a fresh DigitalOcean cloud builder (AMD EPYC 8c, Ubuntu 24.04), BOARD=helios64 BRANCH=edge:

# Configuration Runtime Marker
1 USE_CCACHE=yes kernel cold 77:45 min Ccache result hit=12 miss=14410
2 USE_CCACHE=no kernel 55:54 min no ccache markers (as expected)
3 USE_CCACHE=yes uboot+ATF cold 2:11 min Ccache result hit=8 miss=1002
4 ENABLE_EXTENSIONS=ccache-remote + CCACHE_REMOTE_STORAGE=http://m1:8088/ccache/ over WireGuard interrupted (~5 min of compile) Remote ccache result hit=4536 miss=2791 write=2740 err=0 (61%)

Logs: paste.armbian.com/{enaretezuc,noxayoyoni,vonuyajido}.

Known design choices

  • USE_CCACHE=yes in userpatches/lib.config does not auto-enable the
    ccache extension. lib.config is sourced after
    initialize_extension_manager, too late to enable extensions; the
    framework limitation is documented in main-config.sh (too late to define hook functions or add extensions in lib.config). The shim
    emits a warning at compile-vars phase nudging users to migrate to
    ENABLE_EXTENSIONS=ccache (the auto-enable code path).
  • Auto-enable triggers on three signals: USE_CCACHE=yes,
    PRIVATE_CCACHE=yes, or ENABLE_EXTENSIONS=ccache-remote (the only
    extension in-tree that sets USE_CCACHE=yes from its
    extension_prepare_config). Any future extension that also sets
    USE_CCACHE from its prepare hook will need to add itself to the
    same trigger list, since the shim runs before
    initialize_extension_manager.

Summary by CodeRabbit

  • New Features

    • Extensible compile-wrapper that runs pre/post build hooks and guarantees post-hook execution on interruption.
    • New ccache compile-cache backend with shared/private modes, configurable cache dir, optional pre/post-build cache reporting, and umask preservation across clean envs.
    • Automatic activation of the compile-cache backend when requested by config.
  • Refactor

    • Compilation cache handling moved to an extension-based system and integrated with build steps (ATF/Crust/kernel/uboot) while preserving legacy post-build reporting.

Review Change Stack

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59c1420d7e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/functions/compilation/ccache.sh Outdated
Comment thread lib/functions/compilation/compile-wrapper.sh Outdated
@iav iav marked this pull request as draft May 16, 2026 02:45
@iav iav force-pushed the refactor/ccache-as-extension branch from 59c1420 to 29a4bb9 Compare May 16, 2026 04:22
@iav
Copy link
Copy Markdown
Owner Author

iav commented May 16, 2026

@chatgpt review

Sorry for the wrong mention earlier — please re-verify on head 29a4bb9. Variant B rewrite addresses both prior P2 findings:

  1. In-core bridge removed entirely (no more compile_wrapper_pre__ccache_core in lib/functions/compilation/ccache.sh — file is deleted). The ccache hook implementations live in extensions/ccache.sh and fire only when the extension is enabled.
  2. add_cleanup_handler now runs BEFORE call_extension_method 'compile_wrapper_pre' in lib/functions/compilation/compile-wrapper.sh, so a long-lived helper started by one pre hook is cleaned up even if a later pre hook fails.

Four open architectural questions are in the PR description above.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 29a4bb93be

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/functions/configuration/main-config.sh Outdated
Comment thread extensions/ccache.sh
Comment thread extensions/ccache.sh Outdated
Comment thread extensions/ccache.sh Outdated
Comment thread extensions/ccache.sh Outdated
Comment thread extensions/ccache.sh Outdated
@iav
Copy link
Copy Markdown
Owner Author

iav commented May 16, 2026

@codex review

Commit 7 (head 05bf8ee5) addresses your 1 P1 + 5 P2 findings:

P1 'auto-enable after remote': shim in main-config.sh now also triggers on ccache-remote in ENABLE_EXTENSIONS, since that extension sets USE_CCACHE / PRIVATE_CCACHE inside its own extension_prepare_config.

P2 'keep legacy ccache helpers': ccache_get_stat and ccache_hit_pct restored under their original public names so extensions/ccache-remote/ccache-remote.sh's ccache_post_compilation__show_remote_stats keeps working.

P2 'export private CCACHE_DIR' / 'export shared-cache umask': export instead of declare -g. Child shells spawned by run_host_command_logged now inherit CCACHE_DIR / CCACHE_UMASK in env, which is where ccache reads them.

P2 'preserve kernel make PATH prefix': _ext_ccache_prepend_path_in_envs now walks the existing common_make_envs array, finds the PATH= entry placed earlier by run_kernel_make_internal (with the PYTHON3 userbase prefix for dtb tools), and prepends /usr/lib/ccache to it. If no existing entry, falls back to adding one. env -i no longer drops the python tools.

P2 'defer private cache setup until remote config runs': All env wiring (CCACHE_DIR, CCACHE_UMASK, PATH, ${CCACHE}) moved out of extension_prepare_config__ccache into the per-artifact *_make_config__ccache hooks. Those fire at compile phase, after every extension_prepare_config_*, so PRIVATE_CCACHE set late by ccache-remote is visible without any alphabetical-ordering dependency.

Ordering invariant is now documented in the file header.

@iav
Copy link
Copy Markdown
Owner Author

iav commented May 16, 2026

@codex review

(Correcting the mention handle — @chatgpt may or may not work, @codex is confirmed.) Same request as above: please re-verify head 05bf8ee5 against your earlier 1 P1 + 5 P2 findings.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 05bf8ee571

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/ccache.sh Outdated
Comment thread lib/functions/configuration/main-config.sh Outdated
Comment thread lib/functions/configuration/main-config.sh Outdated
@iav
Copy link
Copy Markdown
Owner Author

iav commented May 16, 2026

@codex review

Commit 8 (head 2185ea64) — moved env setup back to the right phase. Your P1 was on the money: *_make_config hooks fire AFTER common_make_params_quoted is built, so ${CCACHE} was empty at substitution time.

Now:

  • Core prepare_compilation_vars exposes a new compile_prepare_vars extension hook. Runs once after every extension_prepare_config_* (so PRIVATE_CCACHE from ccache-remote is settled) AND after userpatches/lib.config / user_config (so late USE_CCACHE overrides are visible), AND before run_*_make_internal builds its arrays (so ${CCACHE} substitution sees the exported value). Order-independent by construction.
  • compile_prepare_vars__ccache exports CCACHE / PATH (/usr/lib/ccache prefix) / CCACHE_DIR (PRIVATE_CCACHE branch) / CCACHE_UMASK (shared-cache branch). All other code in the extension shrinks accordingly.
  • *_make_config__ccache (kernel, uboot) now ONLY inject CCACHE_UMASK into env-i make envs — kernel/uboot already auto-pass CCACHE_DIR; CCACHE_UMASK is the one variable env -i would otherwise drop. ATF/Crust don't use env -i, so their make_config hooks are gone.
  • Shim in main-config.sh honors the EXT alias by using ${ENABLE_EXTENSIONS:-${EXT:-}}, matching what initialize_extension_manager does internally.
  • The third finding (late USE_CCACHE overrides from userpatches/lib.config) is solved by the move: prepare_compilation_vars runs late enough.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2185ea647c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/functions/configuration/main-config.sh Outdated
Comment thread lib/functions/configuration/compilation-config.sh
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d4a7cdb673

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/functions/configuration/compilation-config.sh Outdated
Comment thread lib/functions/configuration/main-config.sh Outdated
Comment thread extensions/ccache.sh
@chatgpt-codex-connector
Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@iav iav force-pushed the refactor/ccache-as-extension branch from f4f17e7 to f32ac58 Compare May 17, 2026 00:41
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f32ac58965

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/functions/configuration/compilation-config.sh
@iav iav changed the title refactor(compilation): extract ccache as extension (prep) + add compile_wrapper hooks refactor(compilation): extract ccache into a swappable extension May 17, 2026
@iav
Copy link
Copy Markdown
Owner Author

iav commented May 17, 2026

@codex review

@chatgpt-codex-connector
Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@iav iav force-pushed the refactor/ccache-as-extension branch 3 times, most recently from f9a751c to d3170e8 Compare May 17, 2026 02:07
iav added 4 commits May 17, 2026 22:43
The variable has been declared but never assigned or used since
8278dc5 (2023-06-20, 'allwinner: Enable crust compilation'), which
copy-pasted the binutils_version + binutils_flags skeleton from atf.sh
but did not bring along the gcc/ld probe block that conditionally
populates the flag, nor the TF_LDFLAGS-style use site. Crust is or1k-elf
firmware where --no-warn-rwx-segment is not relevant, so the absence
of the probe is by design; the leftover empty declaration is just dead.

Removes a pre-existing SC2034 'appears unused' warning that surfaced
when the file is touched by other refactors.

Assisted-by: Claude:claude-opus-4.7
CROSS_COMPILE and CC were hard-coded to 'ccache' which:
  - breaks when USE_CCACHE=no (CCACHE='' but 'ccache' still tried)
  - prevents wrapper substitution by extensions (sccache, distcc)

Match the established kernel/uboot convention where the cache wrapper
is parameterised via ${CCACHE} (set by configure-ccache based on
USE_CCACHE). Default value '' means no wrapper.

Assisted-by: Claude:claude-opus-4.7
…le_wrapper

Adds three new extension points to the compilation pipeline:

  - atf_make_config: called right before invoking 'make' for ATF (TF-A).
    Receives env that will be passed to make; extensions can mutate
    CCACHE / CC / CROSS_COMPILE to inject wrappers.
  - crust_make_config: same for the or1k Crust firmware (defconfig +
    target make).
  - do_with_compile_wrapper / compile_wrapper_pre / compile_wrapper_post:
    high-level wrapper that bookends an arbitrary compilation block
    with pre/post hooks. Lets extensions observe a compile pass
    end-to-end (clear stats / dump stats / mount caches / etc) without
    having to patch each artifact's compile.sh.

Pure addition: no existing call site is wired yet (kernel/uboot
migration follows in subsequent commits). The hooks are no-ops until
an extension implements them.

Assisted-by: Claude:claude-opus-4.7
Extracts ccache-specific logic from lib/functions/compilation/ccache.sh
into a new opt-out extension extensions/ccache.sh. The extension wires
itself into the compilation pipeline through the new hook points:

  - compile_prepare_vars: dispatched from prepare_compilation_vars
    (early). Computes CCACHE / CCACHE_DIR / CCACHE_UMASK / PATH and
    exports them so subsequent make-quoted-params arrays expand
    correctly. Runs before any artifact's make invocation.
  - compile_wrapper_pre/_post: zeros / reports per-build stats around
    each artifact's compile pass.
  - {kernel,uboot,atf,crust}_make_config: ensures CCACHE_DIR /
    CCACHE_UMASK are visible to the make subshell on each artifact.

This is an additive change: nothing yet calls call_extension_method
"compile_prepare_vars" — that wiring lands in the next refactor commit.

Assisted-by: Claude:claude-opus-4.7
…che.sh

Wires the new ccache extension (extensions/ccache.sh) into the
compilation pipeline, replacing the in-core lib/functions/compilation/
ccache.sh module that was sourced unconditionally.

Changes:
  - lib/functions/compilation/kernel.sh, uboot.sh: wrap the compile
    pass with do_with_compile_wrapper so the new compile_wrapper_pre/
    _post hooks fire around it.
  - lib/functions/configuration/compilation-config.sh: dispatch
    compile_prepare_vars (early phase, before any make-quoted-params
    array is built) instead of inlining configure-ccache. Also clears
    stale CCACHE export from a previous artifact iteration so a
    second uboot/kernel call in the same shell can't reuse a now-
    disabled cache. Keeps a one-shot deprecation warning if any
    user-patch still sources the legacy lib.config.
  - lib/functions/configuration/main-config.sh: if USE_CCACHE=yes
    (or the ccache-remote extension is enabled) auto-enable the
    'ccache' extension before initialize_extension_manager runs.
    Honours both EXT alias and the legacy space-separated
    ENABLE_EXTENSIONS form.
  - lib/functions/general/extensions.sh: extract
    extension_list_normalized helper used by the auto-enable shim.
  - lib/functions/compilation/ccache.sh: removed (logic moved to
    extensions/ccache.sh).

Behaviour is preserved for USE_CCACHE=yes (default-on for kernel/uboot
builds) and USE_CCACHE=no callers. ENABLE_EXTENSIONS=ccache-remote
implicitly pulls in ccache, matching pre-refactor coupling.

Assisted-by: Claude:claude-opus-4.7
@iav iav force-pushed the refactor/ccache-as-extension branch from d3170e8 to 6dd6a7c Compare May 17, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant